DOCUMENT RESUME 



ED 433 775 



HE 032 325 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Ammerman, Peter A.; McLaughlin, Gerald W.; Brozovsky, Paul 
V. 

Assessing the Effect of Programs Designed To Enhance Student 
Success. AIR 1999 Annual Forum Paper. 

1999-06-00 

29p.; Paper presented at the Annual Forum of the Association 
for Institutional Research (39th, Seattle, WA, May 30- June 
3, 1999) . Document contains light type and small print that 
may not reproduce clearly. 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

Academic Achievement; Dropout Prevention; *Grade Point 
Average; *High Risk Students; Higher Education; 

Intervention; Models; *Program Effectiveness; *Program 
Evaluation; School Holding Power; Statistical Analysis; 
Student Development; Student Participation 
*AIR Forum 



ABSTRACT 



This study evaluated the effectiveness of a variety of 
academic intervention programs designed to improve the success of "at risk" 
students at one university. Most programs were designed to either help 
students in a particular course (such as calculus or chemistry) or in a 
particular major. The performance benchmark used to assess program 
effectiveness was Spring 1997 semester grade point average in comparison with 
predicted grades based on a linear regression statistical model. Analysis 
indicated that the 13 academic intervention programs were generally effective 
in improving student performance as measured by the Spring semester GPA. 

Among four programs which appeared to be particularly effective were a 
program for forestry students and a program for students taking chemistry. 

Two programs, one for African-American students and one for at-risk biology 
majors, appeared to have a slight detrimental effect on spring grades. 

Further analysis suggested that the positive results for the effectiveness of 
the intervention programs seemed to be driven largely by the results for 
those students who participated in more than one program. (Contains 18 
references.) (DB) 



******************************************************************************** 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document . * 

******************************************************************************** 




g Assessing The Effect Of Programs Designed To Enhance 
g Stydent Success 



By 

Peter A. Ammerman, 

Institutional Research and Planning Analysis, Virginia Tech 
Gerald W. McLaughlin, 

Institutional Planning and Research, DePaul Univeristy 

And 

Paul V. Brozovsky 

Institutional Research and Planning Analysis, Virginia Tech 







U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
y CENTER (ERIC) 

^This document has been reproduced as 
received from the person or organization 
originating it. 

D Minor changes have been made to 
improve reproduction quality. 



® Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 

. . 



AIR Forum 
May 30-June 2, 1999 
Seattle, Washington 



BEST COPY AVAILABLE 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

D. Vura 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




for Management Research, Policy Analysis, and Planning 



This paper was presented at the Thirty-Ninth Annual Forum 
of the Association for Institutional Research held in Seattle, 
Washington, May 30-June 3, 1999. 

This paper was reviewed by the AIR Forum Publications 
Committee and was judged to be of high quality and of 
interest to others concerned with the research of higher 
education. It has therefore been selected to be included in 
the ERIC Collection of AIR Forum Papers. 



Dolores Vura 
Editor 

Air Forum Publications 



ERIC 



3 



Assessing The Effect Of Programs Designed To Enhance Student Success 



Abstract 

When we implemented a new academic eligibility requirement, we also initiated some thirteen programs to improve 
student success. To evaluate the programs, we used a methodology to make comparisons across various programs 
ranging from those designed to build life skills to those designed to improve grades in specific courses. The basis of 
our methodology is to develop linear models to anticipate performance and then interpret the variables associated 
with various program characteristics. We discuss interpretation of results it terms of both methodological and 
programmatic issues along with next steps in developing the student programs. 
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Introduction and Background: 



A student’s lack of success and consequent dropping out from a higher educational institution can have costly 
consequences for both the student and the institution. For the student, a lack of success in college can impose both 
personal and financial costs, including such factors as lower self-esteem and more limited opportunities for self and 
career advancement. For the educational institution, on the other hand, the costs of student attrition include the 
intangible humanitarian costs associated with failing to help students live up to their full potential. There are also the 
tangible financial costs associated with trying to recruit new students, hence new sources of tuition income for the 
school, to replace the students lost through attrition. 

As a result of these potentially very large tangible and intangible costs of student attrition, the question of how to 
retain students and, in general, to enhance student success at higher educational institutions has been a widely 
studied one over the past few decades. Even before 1980, this problem had received sufficient attention to merit a 
number of comprehensive literature reviews, including those of Spady (1970), Tinto (1975), Cope and Hannah 
(1975), and Pantages and Creedon (1978). Most of the earlier studies comprising this body of literature, however, 
had been atheoretical in terms of their descriptions of the influences on student attrition. So, starting with Spady 
(1971), a number of authors have proposed theoretical models to explain student attrition (e.g., Pascarella (1980) 
and Bean (1986)). However, probably the best known and most widely examined, tested, and accepted model of 
student attrition in the educational community is that of Tinto (1975, 1987, 1993) (see, e.g., Halpin (1990), 

Pascarella and Chapman (1983), Pascarella, Duby, and Iverson (1983), Terenzini and Pascarella (1977,1980), and 
Terenzini, Lorang, and Pascarella (1981)). 

Tinto’s model focuses on the fit between the student and the educational institution and emphasizes integration and 
commitment. According to this model, an individual’s pre-entry attributes, such as family background, skill and 
ability, and prior schooling, interact with each other. They then influence the student’s goals and commitments, most 
notably the student’s initial commitment to the goal of college completion and initial commitment to the institution 
itself. These goals and commitments, in turn, interact overtime with the student’s academic and institutional 
experiences, in terms of both the formal and infonnal academic and social systems of the institution, to influence 
students’ intellectual development and academic performance, which determine academic integration. The extent to 
which the student becomes academically and socially integrated into the formal and informal academic and social 
systems of an institution then determines the individual student’s departure decision. A greater degree of goal and 
institutional commitment corresponds to a reduced probability of the student’s dropping out. 

Thus, in order to improve retention and reduce attrition, an institution should invest its time and energy on 
strengthening those factors that help improve the fit between the student and the institution. This is the role that 
programs aimed at improving student retention need to play. So, on the road to improving student retention, it is 
important first to develop such programs and then to ensure that such programs are performing their function 
effectively. As Congos and Schoeps (1997) state. 

Retention is an outcome, a result, a by-product of effective educational programs and services in and out of 
the classroom. Therefore, focus in postsecondary education should not be on retention itself but rather on 
the effectiveness of programs aimed at lowering attrition. 

We have followed their advice and evaluating program effectiveness is the focus of this paper. We have included 
special emphasis on exploring a number of statistical and analytical issues that arise in trying to assess the 
effectiveness of such programs in the non-experimental environment in which they must operate. 

Building on some of the insights from the vast body of research on student attrition and performance, and in 
conjunction with the development of a new academic eligibility policy, various projects were funded to investigate 
factors that seem to enhance academic success. Felicia Blanks, coordinator of these projects, asked IR to look at the 
effectiveness of these various intervention programs. 

The academic intervention programs that will be studied in this paper were conducted during the Spring semester of 
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the 1996-97 academic year. These programs were designed to help improve the academic success of students who 
were "at risk" in various regards. Some of the programs were designed to help students in a particular course, while 
others were not course-specific but were designed to help students of a particular major. Still other programs were 
designed to help students who were thought to be at-risk for other reasons that did not fall into these two specific 
categories. A listing of the programs analyzed (along with the abbreviations used to denote the programs in the 
analysis below) is as follows: 

» "Excel above and beyond - a comprehensive intervention program for African-American Students" (CAAE) 

“ "Promoting Student Success in Calculus-Based Introductory Physics" (CALPHYSC) 

" "Early Involvement with At Risk Biology Majors" (ERLINVBl) 

“ "The Emerging Scholars Program in Calculus (Math)" (ESPMATH) 

® "Enhancing Success and Professional Development in Forestry" (PRODEVFO) 

• "Fostering Student Success through Residential Advisors" (RESADADS) 

• "Enhancing Student Success - High Risk Courses - Chemistry" (RSKCHEM) 

• "Course-specific Moderated Study Groups to Increase Plan for African-American Students" (STDYUHSA) 

• "Multifaceted Approaches to Increasing Retention in FCD" (SUCCFCD) 

• "Developing Complete Student Through Team Advising - Biology" (TMADVBIO) 

• "Training Project Success Facilitators (CAAE)" (TNGSUFAC) 

The research goals for this study are twofold. First, the general effectiveness of the intervention system as a whole 
will be analyzed. The goal of this analysis will be to determine whether the performance of students who 
participated in a program improved relative to what it would have been if the student had not been a participant in 
the intervention system, regardless of the specific program or programs in which the student participated. Second, 
the performance of specific intervention programs will be examined. 



Criteria For Analyzing The Programs: 

There are multiple ways to evaluate improved performance. The intent of this effort is to select a criterion that is 
relevant to all programs. As was noted above, some of the intervention programs focus on specific courses (e.g., 
"Promoting Student Success in Calculus-Based Introductory Physics"). Other programs focus on specific majors 
(e.g., "Enhancing Success and Professional Development in Forestry"). Other programs target specific students or 
categories of students who want to improve their academic performance (e.g., "Excel above and beyond - a 
comprehensive intervention program for African-American Students" and "Fostering Student Success through 
Residential Advisors"). For the programs that are connected with a specific course, the effectiveness of the program 
could be assessed by comparing the students’ actual performance in the course (e.g., letter or numerical grade) with 
the performance that would have been expected for the students had they not participated in the intervention 
program. For the programs for specific majors, the students’ retention rate within the major and their subsequent 
Major GPA's could be used in evaluating the effectiveness of the programs. 

However, none of these evaluation methods is unilaterally appropriate for all of the programs. Thus, a measure of 
performance must be chosen that IS applicable to the students in all of the programs. While specific programs seek 
to improve student performance in specific categories, frorn a university-wide perspective the purpose of the 
intervention process (as opposed to a specific intervention program) is ultimately to improve the chances that a 
student will (be able to) remain enrolled in school and eventually graduate. For the purposes of short-term 
evaluation, one performance benchmark students establish along the way toward remaining in school and ultimately 
graduating is the GPA they attain for the Spring term, a decisive factor in determining which students will be 
allowed to return in the Fall. 

Thus, the prime performance benchmark that will be used to assess the effectiveness of the intervention programs is 
the Spring '97 semester GPA. But to use this as a performance benchmark for the programs means that we need to 
assess the difference between what are the students' performances, i.e., their Spring GPA's, given that they were 
involved in the intervention programs and what the Spring GPA's WOULD HAVE BEEN had they not been 
involved in the intervention program. This means that a model must first be developed for predicting students' 
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Spring semester GPA*s before it will be possible to determine whether the intervention programs were successful in 
improving Spring semester GPA’s. Of course, once such a model is in place, it could also be used to directly 
examine the effectiveness of the intervention programs in improving students’ Spring semester GPA's. 



Initial Data Analysis: 

Before a statistical model for Spring GPA’s can be developed that will enable an examination of the effectiveness of 
the intervention programs, however, the statistical features of the data must first be examined to determine what 
form such a model should take. Also, a set of possible independent variables must be selected that are likely to be 
able to explain or "capture" the observed variation in the dependent variable, the Spring GPA. 

For GPA data, the most obvious possible explanatory data is past GPA data. Thus, an initial possible model entails 
modeling Spring GPA as a function of Cumulative GPA as of the end of the Fall term. For estimating the 
relationship between past GPA and present GPA, the relevant population is the entire student body. The ultimate 
purpose of our analysis, however, is to estimate the effectiveness of the intervention programs in improving student 
GPA's. Thus, to estimate how well the students may have performed had they not been involved in the intervention 
programs, it is necessary to find a control group of students not in the intervention programs that is as similar as 
possible to the treatment group of students who were in intervention programs. Because the ultimate purpose of the 
intervention programs is to improve student retention and ultimately graduation rates, we are most concerned at this 
point with students who have not yet reached their Senior year. Therefore, our relevant population of interest 
consists of Freshmen, Sophomores, and Juniors, but excludes Seniors. Out of the categories remaining in the 
population. Freshmen are ultimately the most important, and this group will also be analyzed separately. 

For the complete analysis group of interest, including the Freshman, Sophomore, and Junior level students, the 
distributions of Spring '97 semester GPA's and Fall '96 cumulative GPA's each cover the full spectrum of possible 
GPA’s from 0.0 to 4.0, each has a mean and median around 2.5, and each of the two distributions exhibits a slight 
degree of negative skewness. Furthermore, there is a slight amount of truncation at the 4.0 boundary, and a slight 
degree of trailing or tapering off toward the 0.0 boundary, but with a "stacking" of a large number of observations at 
0.0, especially for the Fall 1996 data. However, apart from the truncation at the upper end and the "stacking" at the 
lower end of the range of possible values, the data fall close to having a symmetric, bell-shaped distribution. In this 
regard, the stacking of observations at 0.0 is particularly troublesome. 

Apart from this potential problem of "non-normality," this GPA data also presents another possible problem in terms 
of its modeling, a problem that is also likely to be at least partially caused by the bounded nature of such data. This 
problem is one of a "nonlinear" relationship between the Fall and the Spring GPA’s. A review of Fall vs. Spring 
GPA’s indicated that as we move along the range of people with Fall '96 GPA's from 0.0 to 4.0, their average Spring 
GPA’s start off at about 1 .4, then gradually fall to a minimum of about 1 .2 for people who received around a 0.3 Fall 
GPA, then the average Spring GPA’s start to rise again, staying above the Fall GPA's until around the 2.25 to 2.5 
range. For students with Fall GPA’s above 2.5, their average Spring GPA's all fall below their Fall GPA’s. The 
bounded nature of the GPA data may explain some of this observed behavior, because students who receive around 
a 2.0 in the Fall could easily do either better or worse in the Spring, depending on the relative ease or difficulty of 
the specific courses they take in the Spring, while a student whose cumulative GPA after the Fall was 0.0 would 
have nowhere the go but up (and of course would have a definite incentive to improve), while a student with a 
cumulative GPA of 4.0 in the Fall would have nowhere to go, gradewise, but down. 

To simultaneously compensate for these two problems of non-normality and non-linearity, the students who had 0.0 
cumulative GPA’s as of the end of the Fall tenn were dropped from the sample, and squared GPA terms were 
calculated for use in the final model. The removal of the 0.0 GPA students from our sample reduced our “control” 
group by 379 students, down to 12,440 students, and reduced the “treatment” group by 54 students, down to 83 1 
students. Aside from the empirical justification of trying to render the data set more "normal" by removing 
"outlying" observations, an additional Justification for deleting students with GPA’s of 0.0 from the sample is that 
such students are much more likely than other students to have been affected by such student-specific circumstances 
as deaths in the family, personal illnesses, etc., that cannot be captured adequately (mainly due to lack of any 
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objective information about such factors) by any explanatory or "dummy’* variables. They might also be an artifact 
of a census date for the database that does not coincide with various registrar and fee payment time lines. Thus, the 
circumstances faced by such students could be systematically different from those of other students, in which case 
their exclusion from the analysis would be appropriate for obtaining unbiased results. It is important to note that the 
previously mentioned nonlinear relationship between Fall and Spring GPA’s was not a function of this stacking of 
Fall GPA observations at 0.0. For the subsequent sample excluding the students with 0.0 Fall GPA’s, the nonlinear 
relationship between Fall and Spring GPA’s was still clearly evident. 



Regression Analysis Of Program Effectiveness: 

A natural first thought when studying the effects of various treatments is to try to view the problem from an 
experimental design/ ANOVA framework. However, there are clear ethical questions to be raised if students on the 
verge of failing out of school were to be randomly assigned not to be allowed into programs that may enable them to 
improve their academic performance and remain in school. Thus, as a consequence, there was not a random 
assignment of students to treatment versus control groups in our case, so we do not have an experimental design 
situation. As an alternative, the data for the students is analyzed within a regression framework. 

The dependent variable for our regression analysis, our primary variable of interest, is the Spring ’97 GPA. Our 
ultimate goal is to determine whether a student’s being in an intervention program led to an increase in that student’s 
GPA over what he would have received had he not been in the program. Boxplots of Spring '97 GPA’s for the 
students in our sample, broken down into treatment vs. non-treatment, and freshman vs. non-freshman students, is 
shown in Figure 1 . 



Figure 1 about here 



As can readily be seen, the students in each of the categories shown exhibit similar levels of variability (so that the 
use of ANOVA or regression analysis is not precluded on these grounds). However, there do seem to be systematic 
differences between these categories of students. The means of these categories (denoted by "+’’) appear to drop 
steadily as we move across these categories. Thus; the average Spring ’97 GPA is lowest for Freshman students who 
were involved in an intervention program. The second lowest average was among post-Freshman students involved 
in an intervention program. The third lowest average belonged to Freshman students who were NOT involved in an 
intervention program, and the group with the highest average was post-Freshman students who weren't involved in 
an intervention program. 

At first glance, these results would appear to show that involvement in an intervention program was detrimental to a 
student's grades. However, such a conclusion ignores the systematic differences that exist between the two groups, 
notably the fact that the students selected for the intervention programs are those who are, in general, encountering 
greater than average difficulties in their academic careers. Thus, all else equal, these students would also be 
expected to post lower Spring GPA's. So the real question is not how well the intervention program students did 
relative to the non-intervention students, but how well they did relative to their own, lower, expected Spring GPA’s. 

But to estimate this statistically, as many as possible of the remaining systematic differences between the treatment 
group (intervention program students) and the control group must be accounted for. For example, one systematic 
difference between the two groups is that the treatment group, containing a total of 83 1 students, is comprised of 
72.1% Freshmen, while the control group of 12,440 students comprises only 32.7% Freshmen (see Table 1). As the 
above box-plots illustrated, the average GPA’s for Freshman students tend to be lower than those of other students 
are. A survivorship bias among post-Freshman students helps to account for some of the difference. Thus, the high 
composition of Freshman students in the treatment group would further contribute to this group’s having a lower 
expected Spring GPA relative to that of the control group. These results are shown in Table 1 . 



Table 1 about here 
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Additional variables across which the treatment group could systematically differ from the control group include a 
student's field of study (e.g., engineering vs. non-engineering), gender, and ethnic background. Given that courses 
required in engineering have a reputation of being more difficult than other courses, engineering majors may be 
expected to have lower GPA's than students with the same academic ability who are in other majors. Thus, the 
relative proportions of engineering majors in the groups would affect their relative expected GPA's. It is unclear 
what systematic effects the other categorical variables may have on the GPA's for the groups, but it is very likely for 
the treatment and control groups to differ across at least some of them. For example, a number of the intervention 
programs are designed specifically for African-American students, so that the treatment groups for those programs 
contain a much higher proportion of African-American students than do the other treatment groups or the control 
group. 

Taking all the above into account, a linear regression model was estimated for the Spring '97 GPA data using 
Freshman and non-Freshman GPA's and squared GPA's and various categorical variables (for ethnicity, gender, field 
of study, etc.) as the independent variables, and a backward selection method (with a cutoff of .10) was used to 
remove the variables that did not seem to play a significant role in accounting for the variation in Spring GPA's. 

This process resulted in a final model for the full sample with an analysis of variance and model parameters as 
shown below: This model is shown in Table 2. 



Table 2 about here 



The R-squared of .4565 for this model indicates that the dependent variables remaining in the model can account for 
close to half of the variation in Spring 1997 semester GPA's. Thus, while there is a lot of variation in such GPA's 
that is left unexplained, a substantial proportion (nearly half) of the variation nonetheless can be accounted for. And 
the variables remaining that play a significant role in helping to explain this variation are a student’s Fall 1996 
cumulative GPA (GPA) and squared GPA (GPA_2), the interaction between whether the student is a Freshman and 
his GPA (GPA*FR) and squared GPA (GPA*FR 2), whether or not the student is an engineering major (ENGG), 
whether or not the student is female (FEMALE), whether or not the student is an African-American (AFR1_AM), 
and last, but most importantly for this study, whether or not the student is involved in an intervention program 
(D TRT) and the interaction between this involvement and the student’s GPA (GPA*TRT). 

As would be expected, the coefficient for GPA is significant and positive, so that higher Fall cumulative GPA’s are 
associated with higher Spring semester GPA’s. Of course, due to variations in course loads, workloads, and levels of 
student motivation over time, the relationship between past and present GPA’s is less than 1:1, but the relationship is 
nonetheless sizable and significant. As the saying goes, “the past is prologue,” and the track record a student 
establishes via a cumulative GPA is a significant predictor of the student’s future performance. For Freshmen, 
however, the track record reflected in the cumulative GPA represents a much smaller body of work, in most cases 
only one semester’s worth of grades, so we would expect such “cumulative” GPA’s to have less ability to predict 
future semester GPA’s for Freshmen than for other students. This is precisely what the negative coefficient for 
GPA*FR tells us is the case. This means that the impact of Fall cumulative grades on Spring semester grades is less 
for Freshmen than for non-Freshman students. For non-Freshman students, the net coefficient (bQp^^) for the 

relationship between Fall cumulative GPA’s and Spring semester GPA’s is 0.554659. For Freshmen, the net 
coefficient (bQp^ plus bQpy^*pj^) is 0.429341 (0.554659 - 0.125318). Thus, given its negative sign, the coefficient 

for GPA*FR represents somewhat of a “de-linkage” or “de-coupling” of Spring GPA’s from Fall GPA’s for 
Freshmen as compared to non-Freshmen. 

The fact that the squared GPA terms (GPA_2 and GPA*FR_2) have significant coefficients indicates that, as the 
analysis of the previous section noted, there is in fact a nonlinear relationship between past and future GPA’s. The 
positive signs of these coefficients indicates that there is a premium for stronger past performance, with this 
premium being nearly 50% greater for Freshmen as compared to non-Freshmen. This, when combined with the 
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results for GPA and GPA*FR, would seem to indicate that for Freshmen as compared to post-Freshman students,, 
poor past performance is more likely to be reversed while good past performance is more likely to be repeated. Or, 
the past is more likely to be prologue for Freshmen if they went ahead and performed well in the first place. But, for 
all students, the significance of the squared GPA term indicates that the influence of past grades on future grade 
performance is stronger for higher GPA’s than for lower GPA’s. 

With regard to the remaining non-treatment-effect variables in the model, majoring in engineering would reduce a 
student’s predicted GPA by 0.066145, while being a female rather than a male would improve the student’s 
predicted performance by 0.094505 points. Hispanics and other ethnic groups were not found to have significantly 
different Spring GPA’s than whites, but the story is different for African-American students. All else equal, the fact 
that a student is an African-American would lower the predicted Spring GPA by 0. 197442. Thus, even if they 
started from the same Fall cumulative GPA, an African-American student would have a predicted Spring semester 
GPA that is 0. 197442 points lower than that of a non-African-American student. But, beyond this, the mean and 
median Fall cumulative GPA’s for African-American students are each approximately 0.4 points below those for 
non-African-American students, so the typical predicted Spring semester GPA difference would be even greater than 
that suggested by the AFRI_AM coefficient. This result provides additional Justification for the existence of 
intervention programs specifically targeting African-American students. Note, however, that caution must be taken 
in the detailed interpretation of individual weights because of the multicollinearity among the variables in the model. 

The remaining variables in the model are the variables designed to capture the treatment effects, D_TRT and 
GPA*TRT; Notably, the parameter estimate is positive for D_TRT and negative for GPA*TRT. These are 
precisely the signs we would expect if the treatment programs were having a beneficial impact. The direct effect of 
the academic intervention programs on predicted Spring semester GPA’s is captured by the D TRT variable, the 
coefficient of which indicates that, all else equal, participation in an intervention program will increase a student’s 
predicted Spring semester GPA by 0.239948, or nearly a quarter of a letter grade. But this effect cannot be 
examined in isolation. The GPA*TRT interaction term is designed to capture the indirect effects of participation in 
an intervention program. Similar in effect to the GPA*FR variable, a negative value for the coefficient to 
GPA*TRT represents a “decoupling” of the Spring semester grades from the previous performance of the student as 
the Fall GPA increases. 

For Freshmen, the Fall cumulative grades provide a track record that is too short to provide a good representation of 
the student’s performance. For the intervention program students, on the other hand, this track record may be long 
enough to be representative, but the intervention programs would hopefully change the attitudes of the student 
toward his academic work, so that the past performance would no longer be representative of the student’s current 
attitudes and resulting academic abilities. Thus, a negative value for the coefficient of GPA TRT means that a 
student’s being in an intervention program will reduce the extent to which his predicted Spring semester grades are 
dependent upon his past grades. In other words, the GPA_TRT variable would reflect a “starting over” effect. So, 
taken together, the significant positive coefficient for D_TRT and the significant negative coefficient for GPA_TRT 
indicate that participation in an intervention program reduces the impact of past grades on current GPA and, 
simultaneously, provides a direct improvement to current GPA. In other words, these results indicate that the 
academic intervention programs provide for a statistically significant improvement in students’ academic 
perfoirnance, regardless of their past performance. 

For a quick review of the variables in the model, a brief example is as follows. For a Hispanic male Psychology 
major in his Sophomore year who is not involved in any of the intervention programs, his predicted GPA for Spring 
'97, as a function of his Fall cumulative GPA (GPA), is: 

Predicted Spring GPA = 0.8443 10 + 0.554659*(GPA) + 0.05 1 327*(GPA)2 

If he were a Freshman rather than a Sophomore, the following amount would be ADDED to the above figure to 
obtain the final prediction of the Spring GPA: 




-0.1 253 1 8*(GPA) + 0.024446*(GPA)2 

10 
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Thus, the predicted GPA in this case would be: 

Predicted Spring GPA = 0.844310 + (0.554659 - 0. 1253 18)*(GPA) + (0.051327 + 0.024446)*(GPA)2 



or, 



Predicted Spring GPA = 0.8443 10 + 0.42934 1*(GPA) + 0.075773*(GPA)2 

Once again, the reason for incorporating the squared GPA term is to account for the nonlinear relationship between 
past GPA's and present GPA’s. 

As a final example, if the student were a Freshman, female, engineering major enrolled in one of the programs, her 
predicted Spring semester GPA, as a function of her Fall cumulative GPA, would be: 

Predicted Spring GPA = (0.844310-0.066145+0.094505+0.239948) + (0.554659-0.125318-0.081 1 14)*(GPA) + 

(0.05 1 327+0.024446)*(G PA)2 



or. 



Predicted Spring GPA = 1.1 12618 + 0.348227*(GPA) + 0.075773*(GPA)2 



Next, because Freshmen retention is a particular concern, a separate analysis was undertaken to focus exclusively on 
the Freshman students in the sample. The results of this analysis are shown in Table 3. 



Table 3about here 



This model exhibits an R-squared only slightly higher than that of the model for ail of the students, but the model 
coefficients exhibit some key differences from those of the full sample model. The intercept term is 0.133 points 
higher for the Freshman sample model than for the full sample model, while the GPA term is 0.106 lower than the 
GPA term for the full model, even after adjusting for the GPA*Freshman interaction term. Among the demographic 
variables, each of these has the same sign as for the full sample model, but the magnitudes of their coefficients are 
substantially larger for the Freshman sample model than for the full sample model. Thus, the predicted Spring 
semester GPA’s of Freshman women are better relative to those of Freshman men than were the predicted Spring 
semester GPA's of all the women in the full sample relative to those of ail the men. Unfortunately for Freshman 
men, this disparity is only compounded if they also happen to be African-Americans and/or are majoring in 
engineering. Freshmen students in these two categories seemed to have even lower grades relative to their peers 
than did Junior and Senior students who fell into these categories. Thus, intervention programs tailored toward such 
Freshman students may be particularly beneficial. 

But the most dramatic differences between the two models concern the treatment effects. First, the direct impact of 
the intervention programs on Spring semester GPA’s, the D_TRT term, has a much smaller coefficient in the 
Freshman sample model relative to the full sample model, 0.063346 versus 0.239948. More dramatically, the 
GPA*treatment interaction effect does not even appear in the Freshman sample model (its p-value was 0. 1252, 
above the 0. 10 cutoff for the backward selection). One possible explanation for these surprising results is that they 
are consequence of the fact that the Freshman sample contains a much higher proportion of students involved in the 
intervention programs, so that part of the treatment mean and GPA interaction effects have been subsumed into the 
direct intercept and GPA effects for this sample. As an illustration, the intercept and D TRT coefficients sum to 
1 .040 for the Freshman sample model, and they sum to the similar amount of 1 .084 for the full sample model. But 
the distribution of these sums between the intercept and the D_TRT coefficients are very different in each case of 
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the two cases, with 22% of the sum being allocated to the D_TRT coefficient for the full model but only 6% being 
allocated to the D_TRT coefficient for the Freshman sample model. Similar results hold for the GPA effects. For 
the Freshman sample model, the GPA effect has a coefficient of 0.323, and there is no significant GPA*treatment 
interaction effect. For the full sample model, on the other hand, the net GPA effect for Freshmen in intervention 
programs is 0.348 (0.554659 -0. 1 253 1 8 -0.08 1114). Even without a significant interaction term to “decouple” 
current grade performance from past grade results, the coefficient for the Freshmen-only sample is still somewhat 
lower even than what would be suggested by the full-sample model for treatment-group Freshmen. Thus, it appears 
that, for the Freshman sample, the entire GPA*treatment interaction effect has simply, but completely, been 
subsumed into the direct GPA effect. 

This explanation is lent validity by the changes in results that occur if the GPA*treatment interaction term is left in 
the model, so that this effect is forcefully separated out from the direct GPA effect. In this case, the coefficient for 
the direct GPA effect increases from 0.323 shown in Table 3 to 0.351, with a coefficient of-0.053 for the 
interaction term (with a p-value of 0.1252), for a net GPA effect for Freshmen enrolled in the intervention programs 
of 0.298. Furthermore, and most dramatically, the direct treatment effect in this case nearly triples in magnitude to a 
level of 0.179, a value that is much closer to what would be suggested by the full sample model. 

An alternative explanation for the Freshman model results as presented is that, for Freshman students, the academic 
intervention programs had little marginal effect toward encouraging a higher level of performance among “at-risk” 
students beyond the impetus produced simply by the shock of receiving poor first semester grades. Thus, under this 
account, where the programs would seem to have been particularly helpful would be among the students who were 
able to maintain a sufficiently high GPA to make it past their Freshman year but were not able, on their own, to 
substantially improve their performance. In such cases, the academic intervention programs would have provided 
the students with the skills they needed to improve their performance but were not able simply to acquire or develop 
on their own. 

The above models examine the effect of participation in an academic intervention program on the more immediate 
performance measure of a student’s Spring semester GPA. The next step in our analysis of the programs was to 
examine their effect of the somewhat longer-term measure of whether a student returned to school for the following 
Fall semester. For these models, the coefficients for the treatment effect variables all had the “correct” signs, but 
unfortunately these effects were not of a large enough magnitude to be statistically significant. Thus, the results in 
this direction are encouraging but inconclusive. 



Examination Of The Effects Of Individual Programs On Predicted GPA's: 

The previous section found that participation in an academic intervention program does tend to improve a student’s 
predicted Spring semester GPA. However, because all students who participated in an intervention program were 
simply lumped together into one all-encompassing treatment group in the models used, such analysis implicitly 
assumed that all the programs were equally effective and that there was no marginal benefit to being in more than 
one program. Given the diversity of types and focuses of the programs involved, however, such an assumption of 
homogeneity is probably unrealistic. So in this section, the models of the previous section are re-estimated with 
variables included to capture the effects of the program or programs in which a student participated. 

This first model estimated is the one that contains all the demographic and GPA variables that were in the full 
sample model of the previous section. However, the variables for the direct treatment effect and the GPA*treatment 
interaction effect have been replaced by the concomitant variables for each of the individual programs in which a 
student could have participated. 



Table 4 about here 



Overall, the results for this model are consistent with those of the simple full sample treatment effect model of the 
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previous section. The coefficients of all of the GPA and demographic variables have the same signs and magnitudes 
very close to those of the variables in the original model. Furthermore, for all but two of the intervention program 
groups, the direct and interaction treatment effect variables also have the anticipated signs, as did the concomitant 
variables for the previous model. Thus, after taking into account the direct effects of the Fall cumulative GPA’s, the 
effects of the demographic variables, and the effects of the other intervention programs, most of the programs 
provide a direct boost (positive weight) to the students’ Spring semester GPA’s while reducing the impact (negative 
weight) of the students’ past grades on this measure. 

Of the eleven programs covered by this study, only two, CAAE and ERLfNVBI, had signs implying that 
participation in the program by a student led to poorer performance, and each of these had the “wrong” signs for 
both the direct treatment effect and the GPA*treatment interaction effect. Notably, however, neither of these effects 
was significant for either of these intervention programs. Of the remaining nine intervention programs, all of them 
had the desired signs for both of their effects. But, of these nine, only four had significant direct effects on student 
grades, while, of these four, three also had significant GPA*treatment interaction effects. Overall, for each of the 
programs it is either the case that both the direct and the interaction effects for the program indicate that the given 
program is beneficial to student performance or they both indicate that the program is “detrimental” to student 
performance. But all the programs that had significant effects on students’ predicted performance were programs 
whose effects appeared to be beneficial. Thus, to the extent that two of the programs appeared to have detrimental 
effects on student performance, these effects were of a small enough magnitude to be attributable simply to random 
variation. 

Note, however, that in interpreting the effects of these variables it is important to remember that these are marginal 
effects given other characteristics and other program participation. They do not tell what the effects of these 
programs would be in isolation. Instead, these coefficients tell what the effect of a given variable is given that the 
effects of other relevant variables have been taken into account. Because a number of students participated in more 
than one of the academic intervention programs, this means that the model coefficients shown represent the effects 
that a given program has on a student’s performance after the effects of any other programs in which a student has 
participated have already been taken into account. 

If the variables that do not have significant effects are removed from the model and only variables that have 
significant effects remain, the results are as shown in Table 5: 



Table 5 about here 



In comparison to the previous model in which the effects of all of the intervention programs were included, this 
edited model holds no surprises. All of the non-treatment-effect variables have coefficients very close to those of 
the complete model, and the treatment effect variables that remain in the model are precisely the same ones that 
were significant in the complete model. Namely, PRODEVFO, RSKCHEM, SUCCFCD, and TNGSUFAC each 
provides a significant direct boost to students’ predicted Spring semester GPA’s, while three of these same 
programs, PRODEVFO, RSKCHEM, and TNGSUFAC each also significantly reduces the weight of students’ past 
GPA track record on their Spring GPA performance. 

The next two sets of results are the complete model and edited model results for the Freshman-only sample of 
students. For these models, only one of the treatment programs appears to have reversed signs for its direct and 
GPA interaction effects, namely ERLINVBI. But, as was the case for the full sample model, neither of the 
coefficients for this treatment program is statistically significant. Among the other nine programs (CALPHYSC had 
no Freshman students), only PRODEVFO, RSKCHEM, SUCCFCD, and TNGSUFAC have significant direct effects 
on predicted Spring semester GPA’s. Of these four programs, two, PRODEVFO and TNGSUFAC, also have 
significant GPA interaction effects. These results are shown in Tables 6 and 7. 



Tables 6 and 7 about here 



Thus, for both the full student sample and the Freshman sample, most of the treatment programs seemed to have a 
positive effect on student performance. Only four of these programs, however, PRODEVFO, RSKCHEM, 
SUCCFCD, and TNGSUFAC, have significant treatment effects, but they are significant for both the full sample 
and the Freshman-only sample models. Of these four, PRODEVFO, RSKCHEM, and TOGSUFAC also have 
significant GPA interaction effects for the full sample model, while PRODEVFO and TNGSUFAC have significant 
GPA interaction effects for the Freshman-only sample model. The fact that the Freshman model in this case has 
such significant GPA*treatment interaction terms is most likely a consequence of the fact that we allowed for 
differences between the programs in this case, rather than simply assuming that all programs were equally effective. 
Conversely, the results of this section indicate that it is the marginal effects of these four programs that are driving 
both the direct treatment effect and the GPA* treatment interaction effect for the generic treatment group of the 
previous section’s model. While most of the remaining treatment programs also appear to be have beneficial effects 
on students’ predicted Spring semester GPA’s, the results for these programs were of too small a magnitude to 
preclude the possibility that the direction of these effects was due solely to random variation. 



Analysis of the Total ‘‘Own-Program” Effects of the Individual Intervention Programs: 

The previous section examined the marginal effects of the various academic intervention programs. In other words, 
what the results show us are the effects of each of the programs given that the effects of all the other programs have 
already been taken into account. The purpose of this section, on the other hand, is to try to study the total effects, 
rather than just the marginal effects, of each of the programs. That is, the goal is to see how effective each program 
was on its own, irrespective of the effects of the other programs. If each of the programs had completely separate 
rosters of students enrolled in them, so that the programs were mutually independent in terms of the specific students 
they tried to benefit, then the marginal effects measured in the previous section would be the same as the average 
“own-program” effects, and the results of the previous section would simultaneously answer the question of whether 
these own-program effects are significant. Unfortunately, because there was substantially degree of overlap of 
students across the programs (see Table 8 for a breakdown of students in multiple programs), the results of the 
individual programs are not independent of each other. Thus, there is no statistically valid means of analyzing the 
effects of each program completely independently of those of the other programs. Although there is no statistically 
valid means of segregating the effects of the individual programs from each other in order to accurately examine the 
total rather than the marginal effects of each program, this section will nonetheless look at this question from a 
couple of different directions. 



Table 8 about here 



Before continuing on*to a more formal analysis of the total “own-program” effects of the various academic 
intervention" programs, the individual effects of each of the programs will first be examined informally, via the 
following method. First, the model for predicting Spring semester GPA’s was fitted, but without including any of 
the treatment effect variables, and then the mean levels of the residuals from this model are calculated for each of 
the programs. The results of this step, for both the complete sample and the Freshman-only sample, are shown in 
Table 9. In this case, the residuals for students in multiple programs were simply counted multiply, and were 
included in the results of each of the programs in which the student participated. 



Table 9 about here 



As can be seen from Table 9, only the CAAE students had negative mean residuals for both the complete and the 
Freshman-only samples. This would imply that, on average, students who participated in the CAAE program 
actually did worse than they would have been expected to do in the absence of such a program. In both the 




14 



12 



Freshman-only and complete sample cases, though, the mean level of the residuals is close enough to zero, only 
0.152 standard deviations of the mean away from zero for the total sample case and 0.780 for the Freshman-only 
sample case, so that chance or random variation cannot be ruled out. This is similar to the results of the previous 
section, in which CAAE also yielded negative, though insignificant, marginal effects on students’ predicted GPA’s. 
For the Freshman-only sample, aside from CAAE all of the programs had positive mean residuals, so that they are 
all associated, even if insignificantly or weakly, with better student performance than would otherwise be predicted 
in the absence of such programs. This is even the case for ERLITsIVBI, which is the only program associated with 
detrimental marginal effects in the Freshman-only sample. 

For the complete sample model, the CALPHYSC displayed the poorest results, with actual Spring semester GPA’s 
on average 0.16 points below what would otherwise have been expected. However, due to the small number of 
participants in this program, this result is only 1.22 standard deviations of the mean away from zero, so it is too 
small a result to be conclusive. The ESPMATH and RESADADS programs also have negative mean residuals, 
though the means in these two cases are even closer to zero. At the other end of the performance spectrum, 
PRODEVFO, TNGSUFAC, TMADVBIO, and SUCCFCD all have substantial positive mean residuals, in terms of 
both absolute magnitude and standard deviations of the mean away from zero. Three of these programs, 
PRODEVFO, TNGSUFAC, and SUCCFCD, were also among the four programs with significantly positive 
marginal effects on expected GPA’s, The fourth program that had significantly positive marginal effects was 
RSKCHEM. The mean residuals for this program were somewhat smaller in magnitude than those of the other three 
programs, though they were still fairly large in relation to the standard deviation of the mean for this program. Thus 
far, the overall results for the own-program total effects of each program seem to be similar to those for the marginal 
effects for each program, with positive effects being associated with most of the programs, and the same four 
programs PRODEVFO, RSKCHEM, SUCCFCD, and TNGSUFAC, being associated with the largest relative 
effects in both cases. 

As was noted previously, the above analysis is one attempt to try to capture the “own-program” effects, or the total 
effects on a student’s Spring semester GPA that can be attributed to a given program. One major problem with this 
analysis, however, is that the results for each of the students who participated in multiple programs are included in 
the results for each of the programs in which the student participated. This confounds the results for the individual 
programs. Such a student may in fact have performed well as a consequence of his participation in a given program. 
On the other hand, the student may have derived no benefit whatsoever from that program but instead performed 
well as a result of the influence of another program. In this case, the student’s good performance would tend to 
reflect well on both the program that had a beneficial influence on him as well as the program that did not provide 
him any benefit. Similarly, it may be the case that none of the programs individually provided much benefit to the 
student, but instead it was the constant reinforcement the student faced by virtue of participating in multiple 
programs that led to an improved performance by the student. 

One means to tiy to correct for the impact of this potential effect is to perform the above analysis with all of the 
students who participated in more than one program being separated out into a separate, unique “MULTIPLE” 
category. Thus, for example, a student enrolled in both the CAAE and the TNGSUFAC programs would be 
included in neither of those groups, but would instead be categorized as MULTIPLE category students. Then, all of 
the students who remained in the categories of the various intervention programs would be students who participated 
only in those programs, so that any effects that were reflected in their performance would be attributable largely to 
the program in question, not to any of the other programs. Of course, even then the impact of other programs cannot 
be completely eliminated from the results, due to the fact that the students enrolled in only one of the programs may 
have indirectly gained useful insights from other programs as a result of interaction with their multiple-program 
classmates. Presumably, however, such an effect would be relatively minor. A more serious problem with this 
approach, though, is that the students involved in multiple programs are also much more likely to be the more 
ambitious, more active students who would put more effort into these programs and consequently be more likely to 
benefit from each of them. Excluding such students from the analysis of the individual programs, therefore, would 
deprive these programs of their brightest representatives, so that the perceived performances of these programs 
would be biased downward. 

Ultimately, therefore, there are serious problems associated with including the results for multiple-program students 
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in with the results of the individual programs in which they participated, and there are equally serious problems 
associated with removing them to a separate category. Nonetheless, examining the data from both of these 
approaches will, it is hoped, lead to a better overview of the total and relative levels of effectiveness of each of the 
programs that were funded. Thus, after the multiple-program students have been segregated to a separate 
“MULTIPLE” category, the mean levels of residuals for the students in this category and for the remaining students 
in the individual intervention programs are as shown in Table 10. 



Table 10 about here 



The most notable results resulting from this categorization are that, on average, the students enrolled in multiple 
programs had by far the best performance relative to what would otherwise be expected, and, concomitantly, 
removing these students from the analysis of the individual programs substantially reduces the average results for 
these programs. This effect is particularly detrimental to the results of TNGSUFAC, nearly a third of whose 
students were also involved in other programs (notably CAAE), ERLINVBI, fully half of whose students also 
participated in other programs, and STDYUHSA, which had proportionately fewer multiple-program students, but 
whose mean results nonetheless fell from +0.089 to -0.129 when these students were removed from the analysis. 

The results for the Freshman-only sample are similar to those for the complete sample, though in this case the 
change in results for the TNGSUFAC program when the multiple-program students are excluded are particularly 
dramatic, from an average of +0. 1 83 with the multiple-program students to -0. 1 54 without. Thus, it seems clear 
that, for whatever reason, whatever the program, its most-improved students were those who were also involved in 
other programs. Furthermore, when these students were removed from the analysis of the programs, the remaining 
students of more than half (seven out of eleven) of the programs actually received lower Spring semester GPA’s 
than would have been expected for them in the absence of such programs. This result could be doing no more than 
representing the fact that if you remove the best performers from a group, the group’s average will necessarily fall. 
On the other hand, this may suggest that there was a substantial benefit from the reinforcement that participation in 
multiple programs provides. 

The next step in our analysis is to try to more directly quantify the significance of both the direct effects of each of 
these programs on predicted Spring semester GPA’s and their interactions on the Fall cumulative GPA effect. But, 
as with the above analysis, simply fitting regression models for each of these programs is fraught with 
complications. Because the treatment group’s students were more heavily concentrated toward the lower end of the 
GPA spectrum than was the student body in general, simply deleting all of the treatment group students, save the 
ones in the specific program being examined, from the sample used to fit the regression model would result in a non- 
representative sample, potentially biasing any regression results that are obtained. The other possibility is to keep 
such students in the sample used to fit the model, but then to ignore their programs’ direct and interaction effects in 
fitting the regression model. This, however, would entail ignoring systematic differences among the students, 
creating an omitted- variables bias among the model parameters that are estimated. This latter problem seems to be 
the more serious, so the foimer approaches is taken. 

Thus, a series of regressions was run, each with the same control group and the same basic model form as for the 
regression model used to analyze the general treatment effects. The treatment group in each case, however, 
consisted only of the students within a specific intervention program. Table 1 1 shows the results for the direct 
treatment effect and the GPA* treatment interaction effect model variables for the complete (i.e.. Freshman, 
Sophomore, and Junior) sample models for which all of the students who participated in a given prograrri are 
included in the treatment group for that program’s model, regardless of whether they also participated in other 
intervention programs. Table 12 shows the equivalent results for the Freshman-only sample models. Notably, in 
Table 1 1 the programs that appear to have significant direct effects are TNGSUFAC, PRODEVFO, SUCCFCD, and 
RSKCHEM, the same four programs that had significant direct marginal effects when all were included in the same 
model. Thus it appears that, with the multiple-program students included in the measurement of the total effects, the 
marginal and the total effects of the various programs seem to be roughly equivalent. Notably, these results suggest 
that the TNGSUFAC program is especially beneficial, with a positive direct effect on its students’ Spring semester 
GPA’s of more than a full grade point. For the Freshman-sample models, the results are also similar to those of the 
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model in which the intervention programs are all included simultaneously, though the variables for the individual 
program models seem in general to be somewhat more significant. 



Tables 11 and 12 about here 



Tables 13 and 14 present the regression model results for the data sets with the multiple-program students 
categorized separately from any of the specific individual programs in which they participated. These results 
contain little surprise. Among both the complete and the Freshman-only sample models the “MULTIPLE” category 
exhibits the strongest treatment effect, in terms of both the direct treatment effect and the GPA*treatment interaction 
effect. None of the other programs, however, exhibits any strongly significant results, either positive or negative. 
These results provide confirmation that it is the students who are involved in more than one program who are 
driving most of the results for each of the individual programs. But even the removal of the multiple-program 
students from their ranks, two of the programs, PRODEVFO and SUCCFCD, still manage to obtain marginally 
significant results, with p-values less than 0.10 for the Freshman, Sophomore, and Junior model. ERLITsIVBI 
exhibits marginally significant results for both the Freshman-Sophomore-Junior model and the Freshman-only 
model, but in this case the results imply that this program is actually marginally detrimental to students’ academic' 
performance. This program exhibits negative performances throughout much of the analysis in this paper, but for 
the most part it is of a small enough maignitude that it can be attributed to random variation. Removing the multiple- 
program students from the analysis, however, actually drops this program to the marginally detrimental range. Of 
course, if the multiple-program students are, in general, the more promising students, then the ERLITsIVBI program, 
half of whose students do participate in multiple programs, would be expected to be quite adversely affected by the 
removal of such students. 



Tables 13 and 14 about here 



Overall, the estimated coefficients for the treatment effects for the individual programs are fairly consistent with 
their mean residuals from the regression models without the treatment effects. Furthermore, when the results for the 
multiple-program students are included with those for each of the programs in which they participate, the results for 
the total effectiveness of these programs is similar to the results for the marginal effects of these programs. 
Unfortunately, there is no way to completely isolate the effects of the programs from each other, so there is no way 
to analyze the effectiveness of the programs in isolation. The key group in causing this statistical complication is 
the subset of students who participated in more than one program, and it is this group per se, rather than any of the 
intervention programs in which they may have participated, that posts the most dramatically positive results, and the 
most dramatic results in general. Separating these students out from the programs in which they participated also 
has a dramatic impact on the posted performances of those programs, so that none of the programs individually 
posted a highly significant performance. In such a case, whether a program’s treatment effect regression 
coefficients have a positive or a negative sign could be simply due to natural random variation among the student 
body, and would thus provide little or no information about whether a program was beneficial or not. The fact that a 
number of programs contained relatively few students means that any positive results achieved by the program could 
easily be overwhelmed by such natural variation. As noted in the previous section, when the students within all the 
programs are combined, the aggregate effect of the programs is significantly positive. But the results of this section, 
in which it is the multiple-program students who seem to be driving most of the results for each of the individual 
intervention programs, raises the question of how much this same category of students is driving these overall 
results. This question will be examined more closely in the next section. 



The Effectiveness of Multiple vs. Single Programs: 

In an earlier section of this paper, it was shown that the intervention programs, taken as a whole, do make a 
significant contribution to the expected Spring semester GPA’s of the students enrolled in these programs. The 
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results of the previous section, meanwhile, indicated that of al l the students in intervention programs, the ones who 
had the best performance were those who participated in more than one program. The results for these students are 
dramatically better than the results for the students enrolled only in one program, while for a majority of these 
programs, the average results for the students in these individual programs are actually worse than what would 
otherwise have been expected in the absence of such programs. This raises the question of whether the positive 
overall results for the treatment programs as a whole are also being driven by these multiple-program students. 

To examine this question, the same general methods as in the previous section were followed. To get a preliminary 
overview of the relative effectiveness of single versus multiple programs, the residuals of the GPA regression model 
without the "treatment" term were examined. The average residuals for each of the resulting three categories, when 
all levels of students are included in the model, are shown in Table 15. 



Table 15 about here 



As the results of this table clearly illustrate, the students in multiple programs exhibited a dramatic improvement in 
performance, while the students enrolled in only a single program exhibited a performance that was, on average, 
negligibly better than the non-treatment group students. Among only the Freshman students, the results were even a 
bit more dramatic, with an even higher average performance of the multiple-program students relative to what was 
expected, and average results for the single-program students that is slightly negative. These results indicate that, all 
else equal, the students who participated in only one program received little if any benefit from these programs 
relative to being in no program at all. The students involved in more than one program, on the other hand, 
performed much better than would otherwise have been expected in the absence of such programs, indicating that 
these students received substantial benefits for their participation. 

To more formally test this conclusion, the regression model from above was refitted, with the treatment group 
dichotomized into single-program students and multiple-program students. The ANOVA tables and estimated 
regression coefficients are shown below for, first, the model fit to all the students and, second, the model fit to only 
the Freshman students. The clear result in both of these cases is that participation in multiple programs led to a 
significant increase in a student’s expected Spring semester GPA of more than eight tenths of a grade point and 
simultaneously also significantly reduced the impact of past grades on the student’s current GPA. Participation in 
only one program, on the other hand, had no significant impact on either students’ expected grades or the impact of 
their past grades on their present grades. These results are shown in Tables 16 and 17. 



Tables 16 and 17 about here 



Thus, the treatment group effect that was reported previously is in reality very heavily a multiple-program-student 
effect, or a multiple-treatment-program effect. In other words, it was the students who participated in multiple 
programs who predominately drove the significant results for the treatment program effects. There are a number of 
possible reasons why this is the case. One possibility is simply that participation in multiple programs provides 
sufficient reinforcement so that the students involved more readily internalize the lessons taught by these programs. 
A variation on this takes note of the fact that students learn in different ways and have poor academic performance 
for a variety of reasons, while each of the individual academic intervention programs must necessarily be designed 
to target student archetypes and reasons for poor performance that seem likely to be representative. Due to the 
natural variation among students, some of the students in a given program will match these target archetypes and 
will benefit accordingly, while other students would not really be a good match for the programs and would find 
them largely irrelevant and providing of little benefit, regardless of how well the individual programs are developed. 
But given that each of the programs targets somewhat different types of students, a student who participates in 
multiple programs is more likely to find at least one program that does match the student’s needs, so that the more 
programs in which the student is involved, the more likely is the student to benefit from at least one of them. 
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The above explanations for the superior performance of students in multiple programs focus on the characteristics of 
the programs themselves. An alternative explanation focuses on the characteristics of the students who participate in 
multiple programs. Namely, the students involved in multiple programs are also much more likely to be the more 
ambitious, more active students who would put more effort into these programs and consequently be more likely to 
benefit from each of them. Thus, under this explanation, the additional benefit that multiple-program students 
receive from the intervention programs relative to that of single-program students and those who did not participate 
in any program is a direct result of the additional work that these multiple-program students put into learning from 
these programs and improving their grades. 

Of course, these explanations are not mutually exclusive, so that each of these factors could come into play. Being 
taught a wider variety of study skills, for example, would make it easier for any student, especially a more actively 
participant one, to find study skills that are well-suited to him, and being more constantly involved in such programs 
would make it even easier for a student who is actively trying to change his academic habits to do so. But while 
these factors can help explain the performance of the students in multiple programs, they do not directly address the 
problem of the apparent lack of beneficial effects of the academic intervention programs among those students who 
were involved in only one program. A partial explanation, though, would follow if in fact the students involved in 
multiple programs were in fact the “best and brightest” among the intervention program students. If these are the 
“upper tail” students of the overall sample, then removing them from the analysis, as separating them out into a 
separate “MULTIPLE” category does, would necessarily reduce the average performance of the remaining students 
who were enrolled only in single programs. Furthermore, the distribution of the GPA performances of these 
remaining single-program students would be negatively skewed, so that the relatively poor performances of the 
“lower tail” students remaining in the sample would have a disproportionate influence on the averages for this 
sample of students. Going beyond this possible simple mathematical explanation, however, would require either 
accumulating additional years’ worth of GPA data about these programs, which would mean delaying any 
conclusions about these programs, or obtaining more detailed information about the specific activities of these 
programs and the students who participated in them. Unfortunately, both of these options are beyond the scope of 
this paper, so that the question of the apparent lack of performance of single-program students must remain 
unresolved. 



Conclusions: 

The purpose of this paper was to study the effectiveness of a variety of academic intervention programs funded by 
the university during the Spring term of 1997. A number of technical issues arose to complicate this study, 
however. One of these was related to the fact that the data available for studying the programs was ultimately 
observational rather than experimental data. An experiment would require a random assignment of the subjects to 
the treatment and the control groups, but in the case of the intervention programs, however, there are clear ethical 
questions to be raised if students on the verge of failing out of school were to be randomly assigned not to be 
allowed into programs that may enable them to improve their academic performance and remain in school. 
Consequently, there was no such random assignment, and we do not have an experimental design/ANOVA 
situation. Instead, the data for these programs were analyzed within a regression framework. Such a framework 
enables us to determine whether the relationship between student performance and participation in the various 
intervention programs is significant, though it does not allow us to make any conclusions about causality. A second 
complicating factor was the fact that the data available for analysis were not normally distributed. The deletion from 
the sample of all students with 0.0 Fall cumulative GPA’s helped to remedy -this problem. A final complicating 
factor was the nonlinear nature of the GPA data. Wliile Fall cumulative GPA’s in general were positively correlated 
with Spring term GPA’s, this relationship did not hold over the entire range of the data. Rather, for Fall GPA’s 
along the range of Just over 0.0 to around 0.3, increases in Fall cumulative GPA’s were actually associated with 
lower Spring term GPA’s. Squared GPA terms were included in the final regression model to help control for this 
effect. 

The results of the regression analysis indicate that the academic intervention programs were effective, as a whole, in 
improving student performance, as measured by Spring semester GPA. When considering all the programs jointly, 
four programs, PRODEVFO, RSKCHEM, SUCCFCD, and TNGSUFAC, were each associated with significantly 
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higher Spring semester GPA’s. Of these four programs, two of them, PRODEVFO and TNGSUFAC, also 
significantly reduce the impact of past grades on the Spring GPA’s, and a third program, RSKCHEM, significantly 
weakens this linkage between past grades and Spring GPA’s for the full sample of students including Freshmen, 
Sophomores, and Juniors, though it did not do so among the Freshman-only sample. Two programs, CAAE and 
ERLINVBI, appeared to detrimentally affect their students’ GPA’s, but the magnitude of this effect was much too 
small to be significant. While the effects of the remaining programs all appeared to be beneficial, these effects were 
also of too small a magnitude to rule out random variation. 

Attempts to isolate the effects of the programs on each other led to similar conclusions about the effectiveness of the 
individual programs. However, another interesting result was revealed by this analysis, namely that the positive 
results for the effectiveness of the intervention programs seem to be driven largely by the results for those students 
who participated in more than one program. When the analysis included only the results for those students who 
participated in only one program, the magnitudes of the effects for the various intervention programs were 
dramatically reduced, to the point where the effects of these programs on students’ GPA’s were no longer 
significant. There are number of possible explanations for the positive results for the multiple-program students, but 
explaining the lack of significance for the single-program students is more difficult. 
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Table 1 



Breakdown of Sample by Intervention Program and Freshman Status 



Program 


Total Students 


Freshmen 


% Freshmen 


CAAE 


128 


64 


50.0 


CALPHYSC 


20 


0 


0.0 


ERLINVBI 


18 


18 


100.0 


ESPMATH 


142 


131 


92.3 


PRODEVFO 


49 


37 


75.5 


RESADADS 


65 


47 


72.3 


RSKCHEM 


217 


168 


77.4 


STDYUHSA 


19 


16 


84.2 


SUCCFCD 


170 


143 


84.1 


TMADVBIO 


33 


30 • 


90.9 


TNGSUFAC 


75 


36 


48.0 


Combined Treatment Group 
Control Group 


831 

12440 


599 

4063 


72.1 

32.7 


Total Sample 


13271 


4662 


35.1 



Note(l) : A number of students were enrolled in multiple intervention programs, so the number of 

students listed in "Combined Treatment Group" is less than the sum of the students enrolled in 
the individual intervention programs. 

Note(2): The "Total Sample" essentially consists of all Freshman, Sophomore, and Junior students 

who had a cumulative Fall 1996 GPA of greater than 0.0. The "Control Group" consists of the 
subset of students in this Total Sample who were not enrolled in any of the academic intervention 
programs . 



Table 2 



Regression 


Analysis 


with Spring 


Semester GPA as the 


Dependent Variable 






Full 


Sample 


Model 








Analysis of Variance 










Sum of 


Mean 






Source 


DF 


Squares 


Square 


F Value 


Prob>F 


Model 


9 


4394 . 62248 488.29139 


1237.415 


0.0001 


Error 


13261 


5232.87036 


0.39461 






C Total 


13270 


9627.49284 









Root 


MSE 


0.62818 


R-square 


0.4565 




Dep 


Mean 


2.62818 


Adj R-sq 


0.4561 




C.V. 




23.90162 












Parameter Estimates 










Parameter 


Standard 


T for HO: 




Variable 


DF 


Estimate 


Error 


Parameter=0 


Prob > |T| 


INTERCEP 


1 


0.844310 


0-.05701596 


14.808 


0.0001 


GPA 


1 


0.554659 


0.04455065 


12.450 


0.0001 


GPA*FR 


1 


-0.125318 


0.02142626 


-5.849 


0.0001 


GPA_2 


1 


0.051327 


0.00876229 


5.858 


0.0001 


GPA*FR_2 


1 


0.024446 


0.00714279 


3.422 


0.0006 


GPA*TRT 


1 


-0.081114 


0.03071281 


-2.641 


0.0083 


D_TRT 


1 


0.239948 


0.07043296 


3.407 


0.0007 


ENGG ■ 


1 


-0.066145 


0.01350847 


-4.897 


0.0001 


FEMALE 


1 


0.094505 


0.01158336 


8.159 


0.0001 


AFRI_AM 


1 


-0. 197442 


0.02762766 


-7.147 


0.0001 
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Table 3 

Regression Analysis with Spring Semester GPA as the Dependent Variable 

Freshman Sample Model 



Analysis of Variance 







Sum of 


Mean 


Source 


DF 


Squares 


Square 


Model 


6 


1796.00977 


299.33496 


Error 


4655 


1810.77564 


0.38900 


C Total 


4661 


3606.78541 




Root MSE 




0.62370 R-square 


Dep Mean 




2.46603 Adj 


R-sq 


C. V. 


25.29142 





F Value 
769.507 



4980 

4973 



Prob>F 

0.0001 



Parameter Estimates 







Parameter 


Standard 


T for HO: 




Variable 


DF 


Estimate 


Error 


Parameter=0 


Prob > |Ti 


INTERCEP 


1 


0.977127 


0.06628763 


14.741 


0.0001 


GPA 


1 


0.323196 


0.05575570 


5.797 


0.0001 


GPA_2 


1 


0.093603 


0.01154666 


8.107 


0.0001 


D_TRT 


1 


0.063346 


0.02849266 


2.223 


0.0262 


ENGG 


1 


-0.093945 


0.02201267 


-4.268 


0.0001 


FEMALE 


1 


0.149471 


0.01950895 


7.662 


0.0001 


AFRI AM 


1 


-0.269764 


0.04456575 


-6.053 


0.0001 



Table 4 

Regression Analysis Allowing for Effects 

Full Sample Model 



of Individual Programs 



Analysis of Variance 



Sum of 



Mean 



Source 




DF Squares Square 


F Value 


Pr< 


Model 




29 4415. 


06267 152.24354 


386, 


.740 


0.( 


Error 




13241 5212. 


43017 0.39366 








Total 




13270 9627. 


49284 








Root 


: MSE 


0.62742 


R-square 


0.4586 






Dep 


Mean 


2.62818 


Adj R-sq 


0.4574 






C. V. 




23.87290 










ater Estimates 
















Parameter 


Standard T 


for HO: 






Variable 


DF 


Estimate 


Error Parameter=0 


Prob > |T| 


INTERCEP 


1 


0.827976 


0.05738526 


14.428 




0.0001 


GPA 


1 


0.565967 


0.04498700 


12.581 




0 . 0001 


GPA_FR 


1 


-0.133450 


0.02148730 


-6.211 




0.0001 


GPA_2 


1 


0.049442 


0.00885515 


5.583 




0.0001 


GPA_FR_2 


1 


0.026911 


0.00715644 


3.760 




0.0002 


GPA_CAAE 


1 


0.025053 


0.13990542 


0.179 




0.8579 


GPA_CALP 


1 


-0.164265 


0.30704092 


-0.535 




0.5927 


GPA_ERLI 


1 


0.271867 


0.19866688 


1.368 




0. 1712 


GPA_ESPM 


1 


-0.086890 


0.05903353 


-1.472 




0.1411 


GPA_PROD 


1 


-0.218837 


0.12050867 


-1.816 




0.0694 


GPA_RESA 


1 


-0.115105 


0.12819734 


-0.898 




0.3693 


GPA_RSKC 


1 


-0.101601 


0.04912197 


-2.068 




0 . 0386 


GPA_STDY 


1 


-0.191315 


0.19844559 


-0.964 




0.3350 


GPA_SUCC 


1 


-0.106399 


0.07723095 


-1.378 




0 . 1683 


GPA_TMAD 


1 


-0 . 153780 


0.18148099 


-0.847 




0.3968 


GPA_TNGS 


1 


-0.429724 


0.18111107 


-2.373 




0.0177 


D_CAAE 


1 


-0.067975 


0.23767676 


-0.286 




0.7749 


D_CALPHY 


1 


0.227911 


0.73737866 


0.309 




0.7573 


D_ERLINV 


1 


-0.465047 


0.39531932 


-1.176 




0.2395 


D_ESPMAT 


1 


0.165367 


0.13316824 


1.242 




0.2143 


D_PRODEV 


1 


0.794993 


0.30049601 


2 . 646 




0.0082 


D_RESADA 


1 


0.241390 


0.31268317 


0.772 




0.4401 


D_RSKCHE 


1 


0.284574 


0.11397057 


2.497 




0.0125 


D_STDYUH 


1 


0.505762 


0.48934578 


1.034 




0.3014 


D_SUCCFC 


1 


0.382406 


0.16200499 


2.360 




0.0183 


D_TMADVB 


1 


0 . 642225 


0.55255680 


1.162 




0.2451 


D_TNGSUF 


1 


0.973258 


0.32320319 


3.011 




0.0026 


ENGG 


1 


-0 . 066060 


0.01362065 


-4.850 




0.0001 


FEMALE 


1 


0.093364 


0.01159612 


8.051 




0.0001 


AFRI AM 


1 


-0 .169722 


0.03045409 


-5.573 




0.0001 



er|c 



24 



22 



Table 5 

Regression Analysis Allowing for Effects of Individual Programs - Edited - 

Full Sample Model 

Analysis of Variance 

Sum of Mean 



Source 
Model 
Error 
C Total 
Root 


; MSE 


DF Squares Square 

14 4409 . 79117 314 . 98508 

13256 5217.70166 0.39361 

13270 9627.49284 

0.62738 R-square 


0. 


F Value 
800.245 

4580 


Prob>F 

0.0001 


Dep 


Mean 


2.62818 


Adj R-sq 


0. 


4575 




C. V. 

Parameter Estimates 
Variable DF 


23.87145 

Parameter 

Estimate 


Standard T for HO: 

Error Parameter=0 Prob > |T| 


INTERCEP 


1 


0.838728 


0.05501853 


•15 


. 244 


0.0001 


GPA 


1 


0.557652 


0.04326031 


12 


.891 


0.0001 


GPA_FR 


1 


-0.129943 


0.02135789 


-6 


.084 


0.0001 


GPA_2 


1 


0.050951 


0.00857139 


5 


. 944 


0.0001 


GPA_FR_2 


1 


0.025643 


0.00711635 


3 


.603 


0.0003 


GPA_PROD 


1 


-0.220835 


0.12037137 


-1 


.835 


0.0666 


GPA_RSKC 


1 


-0.120934 


0.04826922 


-2 


.505 


0.0122 


GPA_TNGS 


1 


-0.444993 


0.17982413 


-2 


.475 


0.0134 


D_PRODEV 


1 


0.799385 


0.30011154 


2 


. 664 


0.0077 


D_RSKCHE 


1 


0.336318 


0.11146961 


3 


.017 


0.0026 


D_SUCCFC 


1 


0.167484 


0.04912896 


3 


.409 


0.0007 


D_TNGSUF 


1 


1.001700 


0.32115931 


3 


. 119 


0.0018 


ENGG 


1 


-0.067281 


0.01351013 


-4 


.980 


0.0001 


FEMALE 


1 


0.093784 


0.01157199 


8 


. 104 


0.0001 


AFRI_AM 


1 


-0 . 176597 


0.02703433 


-6 


.532 


0.0001 



Table 6 

Regression Analysis Allowing for Effects of Individual Programs 

Freshman Sample Model 

Analysis of Variance 







Sum of 


Mean 






Source 


DF 


Squares 


Square 


F Value 


Prob>F 


Model 


25 


1814.30937 


72.57237 


187 . 699 


0.0001 


Error 


4636 


1792.47604 


0.38664 






C Total 


4661 


3606.78541 








Root MSE 




0.62181 R-square 


0.5030 




Dep Mean 




2.46603 Adj 


R-sq 


0.5003 




C . V. 


25.21481 









Parameter Estimates 







Parameter 


Standard 


T for HO: 




Variable 


DF 


Estimate 


Error 


Parameter=0 


Prob > 1 T 1 


INTERCEP 


1 


0.900123 


0.07157409 


12.576 


0.0001 


GPA 


1 


0 . 370129 


0.05931359 


6.240 


0.0001 


GPA_2 


1 


0.087037 


0.01205157 


7.222 


0.0001 


GPA_CAAE 


1 


-0.119168 


0.16343953 


-0.729 


0.4660 


GPA_ERLI 


1 


0.273484 


0.19745845 


1.385 


0.1661 


GPA_ESPM 


1 


-0.078709 


0.05964032 


-1.320 


0.1870 


GPA_PROD 


1, 


-0.224993 


0.13087386 


-1.719 


0.0857 


GPA_RESA 


l' 


-0.236316 


0.14549682 


-1.624 


0.1044 


GPA_RSKC 


1 


-0.059485 


0.05506861 


-1.080 


0.2801 


GPA_STDY 


1 


-0.267652 


0.20787767 


-1.288 


0.1980 


GPA_SUCC 


1 


-0.067302 


0.07942161 


-0 . 847 


0.3968 


GPA_TMAD 


1 


-0. 163175 


o’. 19442854 


-0.839 


0.4014 


GPA_TNGS 


1 


-0.522836 


0.19507281 


-2.680 


0 . 0074 


D_CAAE 


1 


0.065781 


0.26836733 


0.245 


0.8064 


D_ERLINV 


1 


-0.472928 


0.39327320 


-1.203 


0.2292 


D_ESPMAT 


1 


0 . 192384 


0.13644169 


1.410 


0.1586 


D_PRODEV 


1 


0.808430 


0.32218923 


2.509 


0.0121 


D_RESADA 


1 


0.560671 


0.36056977 


1.555 


0.1200 


D RSKCHE 


1 


0.214040 


0.12405646 


1.725 


0.0845 


D STDYUH 


1 


0.788284 


0 . 52773076 


1.494 


0.1353 


D_SUCCFC 


1 


0.282538 


0.16893798 


1.672 


0.0945 


D TMADVB 


1 


0.666949 


0.57903397 


1.152 


0.2495 


D TNGSUF 


1 


1.019574 


0.34296993 


2.973 


0.0030 


ENGG 


1 


-0 . 099477 


0.02228488 


-4.464 


0.0001 


FEMALE 


1 


0.148039 


0.01956389 


7 . 567 


0.0001 


AFRI AM 


1 


-0.217678 


0.05208098 


-4.180 


0.0001 



o 
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Table 7 



Regression Analysis Allowing for Effects of Individual Programs - Edited 









Freshman 


Sample Model 










Analysis of Variance 














Sum of 


Mean 










Source 




DF Squares Square 




F Value 


Prob>F 


Model 




11 1807. 


65747 164. 


33250 




424 , 


.731 


0.0001 


Error 




4650 1799. 


12794 0. 


38691 










C Total 




4661 3606. 


78541 












Root 


MSE 


0.62202 


R- square 




0. 


5012 






Dep 


Mean 


2.46603 


Adj R-sq 




0. 


5000 






C. V. 




25.22350 


















Parameter Estimates 














Parameter 


Standard 


T 


for : 


HO: 






Variable 


DF 


Estimate 


Error 


Parameter=0 


Prob > |T| 


INTERCEP 


1 


0.939088 


0.06618705 




14 


. 188 




0.0001 


GPA 


1 


0.344636 


0.05584173 




6 


. 172 




0.0001 


GPA_2 


1 


0.090737 


0.01155792 




7 


.851 




0.0001 


GPA_PROD 


1 


-0.234245 


0.13059844 




-1 


.794 




0.0729 


GPA_TNGS 


1 


-0.549693 


0. 19328917 




-2 


.844 




0.0045 


D_PRODEV 


1 


0.830251 


0.32147661 




2 


. 583 




0.0098 


D_RSKCHE 


1 


0.108901 


0.04964373 




2 


. 194 




0.0283 


D_SUCCFC 


1 


0.139968 


0.05339758 




2 


. 621 




0.0088 


D_TNGSUF 


1 


1.083095 


0.34037161 




3 


. 182 




0.0015 


ENGG 


1 


-0.094557 


0.02200845 




-4 


.296 




0.0001 


FEMALE 


1 


0. 148507 


0.01946977 




7 


.628 




0.0001 


AFRI AM 


1 


-0.247493 


0.04389627 




-5 


. 638 




0.0001 



Table 8 

Breakdown of Students in Multiple Programs 



Program 


Total Students 


Total in 
Multiple 


Freshmen Freshmen in 

Students Multiple 


CAAE 




128 


24 




64 


15 


CALPHYSC 




20 


1 




0 


0 


ERLINVBI 




18 


9 




18 


9 


ESPMATH 




142 


25 




131 


24 


PRODEVFO 




49 


6 




37 


6 


RESADADS 




65 


20 




47 


16 


RSKCHEM 




217 


47 




168 


44 


STDYUHSA 




19 


5 




16 


4 


SUCCFCD 




170 


31 




143 


30 


TMADVBIO 




33 


5 




30 


5 


TNG SU FAC 




75 


24 




36 


18 


Multiple 




92 


92 




80 


80 








Table 9 










Mean Levels Of Residual GPA’s For Students In Intervention Programs 






All Students 






Freshmen 






Program 


No . Obs . 


Mean 


Std. Dev. 


No . Obs . 


Mean 


Std. Dev. 


All Prgms . 


831 


0.052558 


0.707347 


599 


0.0506726 


0 . 6860604 


PRODEVFO 


49 


0.274730 


0.590868 


' 37 


0.2897860 


0.6442918 


TNG SU FAC 


75 


0.239294 


0.797998 


36 


0. 1830086 


0.7857806 


TMADVBIO 


33 


0.188050 


0.385802 


30 


0 . 1845771 


0.4073902 


SUCCFCD 


170 


0.169997 


0.704626 


143 


0 . 1430916 


0.7100588 


STDYUHSA 


19 


0.089055 


0.716960 


16 


0 . 1577565 


0.7032586 


RSKCHEM 


217 


0.086213 


0.707818 


168 


0 .1199940 


0.6803429 


ERLINVBI 


18 


0.048986 


0.658556 


18 


0.0431744 


0.6625851 


RESADADS 


65 


-0.004820 


0.598953 


47 


0.0170576 


0.5882535 


ESPMATH 


142 


-0.009030 


0.696843 


131 


0.0376169 


0.6700399 


CAAE 


128 


-0.010740 


0.797883 


64 


-0.0792842 


0.8130732 


CALPHYSC 


20 


-0.161520 


0.592037 


0 


n . a . 


n . a . 


CONTROL 


12440 


-0.003511 


0.622434 


4063 


-0.0074706 


0.6136210 


Total Sample 


13271 


0 


0.628205 


4662 


0 


0.6236246 


NOTE : Programs 


are ranked in 


order of their mean residuals from the 


complete sample 


: model. 



o 

ERIC 
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Table 10 



9 

Mean Levels Of Residual GPA’s For Students In Intervention Programs: 
Multiple-Program Students Counted Separately 



Program 


All 
Students 
No . Obs . 


Mean 


Std. Dev. 


Freshmen 
Sample 
No . Obs . 


Mean 


Std. Dev. 


All Prgms. 


831 


0.052558 


0.707347 


599 


0.0506726 


0.6860604 


MULTIPLE 


92 


0.349137 


0.622271 


80 


0.3834308 


0.6115044 


PRODEVFO 


43 


0.215529 


0.554632 


31 


0.2103994 


0.6035106 


TMADVBIO 


28 


0.178926 


0.358225 


25 


0.1763793 


0.3761050 


SUCCFCD 


139 


0.133029 


0.721601 


113 


0.0846861 


0.7290945 


TNGSUFAC 


51 


0.113417 


0.856568 


18 


>0.1536067 


0.8288045 


RSKCHEM 


170 


-0.002440 


0.705365 


124 


0.0099554 


0.6693863 


RESADADS 


45 


-0.048960 


0.644954 


31 


-0.0811828 


0.6332558 


CAAE 


104 


-0.057190 


0.821348 


49 


-0.1758328 


0.. 8396075 


ESPMATH 


117 


-0.106250 


0.662733 


107 


-0.0559626 


0.6326468 


STDYUHSA 


14 


-0.128520 


0.650261 


12 


>0.0510993 


0.6179936 


CALPHYSC 


19 


-0.141000 


0.600914 


0 


n . a . 


n . a . 


ERLINVBI 


9 


-0.168990 


0.584758 


9 


-0.1732081 


0.5920609 


CONTROL 


12440 


-0.003511 


0.622434 


4063 


-0.0074706 


0.6136210 


Total Sample 


13271 


0 


0.628205 


4662 


0 


0.6236246 


NOTE: Programs are ranked in 


. order of their mean residuals from the 


complete sample model. 








Table 11 










Direct and Interaction Effects of Individual Programs on GPA’s 




Program 


b(trt) 


t-stat 


p-value 


b(GPA*trt) 


t-stat 


p-value 


All Prgms. 


0.223053 


3.131 


0.0017 


-0.073889 


-2.380 


0 .0173 


CAAE 


-0.094834 


-0.403 


0.6871 


0.050540 


0.365 


0.7151 


CALPHYSC 


0.235545 


0.322 


. 0.7475 


-0.167578 


-0.550 


0.5823 


ERLINVBI 


-0.460723 


-1.175 


0.2399 


0.281036 


1.427 


0.1537 


ESPMATH 


0.193761 


1.455 


0.1458 


-0.095634 


-1 . 622 


0.1049 


PRODEVFO 


0.833271 


2.796 


0.0052 


-0.231735 


-1.939 


0.0525 


RESADADS 


0.345123 


1.123 


0.2613 


-0.146208 


-1.161 


0.2456 


RSKCHEM 


0.358579 


3.188 


0.0014 


-0.124885 


-2.574 


0.0101 


STDYUHSA 


0.762210 


1.584 


0.1131 


-0.282234 


-1.445 


0.1486 


SUCCFCD 


0.437608 


2.722 


0.0065 


-0.127839 


-1 . 671 


0.0947 


TMADVBIO 


0.735636 


1.348 


0.1778 


-0.183017 


-1.021 


0.3074 


TNGSUFAC 


1.049955 ■ 


3.285 


0.0010 


-0.460033 


-2.570 


0.0102 



Table 12 

Direct and Interaction Effects of Individual Programs on GPA’s for Freshman Students 



Program 


b(trt) 


t-stat 


p-value 


b(GPA*trt) 


t-stat 


p-value 


All Prgms . 


0.239555 


3 .409 


0 . 0007 


-0 . 105336 


-3 . 440 


0 . 0006 


CAAE 


0.070183 


0.265 


0.7907 


-0.120204 


-0.751 


0.4527 


CALPHYSC 


n . a . 


n . a . 


n . a . 


n. a . 


n . a . 


n . a . 


ERLINVBI 


-0.474279 


-1.224 


0.2209 


0.278207 


1.432 


0.1523 


ESPMATH 


0.243467 


1.809 


0.0705 


-0.095372 


-1. 621 


0.1052 


PRODEVFO 


0.867095 


2.729 


0.0064 


-0.243254 


-1.885 


0.0595 


RESADADS 


0.585208 


1.655 


0.0979 


-0.236527 


-1.661 


0.0967 


RSKCHEM 


0.326029 


2.684 


0.0073 


-0.095885 


-1.777 


0.0757 


STDYUHSA 


0.934548 


1.808 


0.0707 


-0.316078 


-1.551 


0.1210 


SUCCFCD 


0.365018 


2.187 


0.0288 


-0.104043 


-1.330 


0.1837 


TMADVBIO 


0.660616 


1.165 


0.2442 


-0.164328 


-0.863 


0.3884 


TNGSUFAC 


1.137292 


3.375 


0.0007 


-0.567556 


-2.967 


0.0030 
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Table 13 



Direct and Interaction Effects of Individual Programs on GPA’s 
Multiple-Program Students Categorized Separately 



Program 


b(trt) 


t-stat 


p-value 


b(GPA*trt) 


t-stat 


p-value 


All Prgms. 


0.223053 


'3.131 


0 . 0017 


-0 . 073889 


-2.380 


0 . 0173 


MULTIPLE 


0.726163 


4.388 


0.0001 


-0.208462 


-2.459 


0.0140 


CAAE 


-0.326830 


-1.269 


0.2043 


0.165448 


1.084 


0.2783 


CALPHYSC 


0.272429 


0.371 


0.7104 


-0.173962 


-0.571 


0.5682 


ERLINVBI 


-0.824623 


-1.715 


0.0864 


0.359280 


1.533 


0.1253 


ESPMATH 


-0.108326 


-0.692 


0.4890 


0.005247 


0.079 


0.9371 


PRODEVFO 


0.555099 


1.673 


0.0944 


-0.137850 


-1.046 


0.2955 


RESADADS 


0.270478 


0.527 


0.5985 


0.124724 


-0.621 


0.5345 


RSKCHEM 


0.114502 


0.855 


0.3924 


-0.047804 


-0.860 


0.3901 


STDYUHSA 


0.195354 


0.295 


0.7683 


-0.130408 


-0.492 


0.6231 


SUCCFCD 


0.363340 


1.881 


0.0600 


-0 . 107641 


-1.171 


0.2418 


TMADVBIO 


0.634502 


0.907 


0.3646 


-0.147505 


-0. 658 


0.5107 


TNGSUFAC 


0.495255 


1.229 


0.2190 


-0.206054 


-0.935 


0.3496 



Table 14 

Direct and Interaction Effects of Individual Programs on GPA’s for Freshman Students 
Multiple-Program Students Categorized Separately 



Program 


b(trt) 


t-stat 


p-value 


b(GPA*trt) 


t-stat 


p-value 


All Prgms . 


0.239555 


3 .409 


0.0007 


-0 . 105336 


-3 .440 


0 . 0006 


MULTIPLE 


0.765262 


4.590 


0.0001 


-0.212598 


-2.525 


0.0116 


CAAE 


-0.141017 


-0.479 


0.6317 


-0.045967 


-0.254 


0.7999 


CALPHYSC 


n . a . 


n . a . 


n . a . 


n . a . 


n . a . 


n . a . 


ERLINVBI 


-0.842471 


-1.776 


0.0758 


0.361065 


1.563 


0.1182 


ESPMATH 


-0 .'049046 


-0.308 


0.7578 


0.002795 


0.042 


0.9665 


PRODEVFO 


0.527631 


1.462 


0.1438 


-0.129979 


-0.897 


0.3698 


RESADADS 


0.528936 


0.770 


0.4411 


-0.235342 


-0.891 


0.3730 


RSKCHEM 


0.009353 


0.063 


0.9496 


0.007019 


0.111 


0.9116 


STDYUHSA 


0.345279 


0.517 


0.6050 


-0.157797 


-0.599 


0.5491 


SUCCFCD 


0.208666 


1.023 


0.3065 


-0.054149 


-0.567 


0.5710 


TMADVBIO 


0.499148 


0.674 


0.5002 


-0.107647 


-0.444 


0.6574 


TNGSUFAC 


0.431898 


0.975 


0.3297 


-0.323030 


-1.373 


0.1697 



Table 15 

Mean Levels Of Residual GPA’s For Students In Single vs. Multiple Intervention Programs: 





All 






Freshmen 








Students 






Sample 






Program 


No . Obs . 


Mean 


Std. Dev. 


No . Obs . 


Mean 


Std. Dev. 


All Prgms . 


831 


0.052558 


0.707347 


599 


0.0506726 


0.6860604 


MULTIPLE 


92 


0.349137 


0.622271 


80 . 


0.3834308 


0 . 6115044 


SINGLE 


739 


0.015636 


0.708969 


519 


-0.0006196 


0 . 6830846 


CONTROL 


12440 


-0.003511 


0.622434 


4063 


-0.0074706 


0.6136210 


Total Sample 


13271 


0 


0.628205 


4662 


0 


0.6236246 



28 




26 



Table 16 



Regression Analysis with Single- vs . Multiple-Program Students 
Complete Sample Model 







Analysis of 


Variance 










Sum of 


Mean 






Source 


DF 


Squares 


Square 


F Value 


Prob>F 


Model 


11 


4405.45715 


400.49610 


1016.879 


0.0001 


Error 


13259 


5222.03569 


0.39385 






C Total 


13270 


9627.49284 








Root MSE 




0.62757 R-square 


0.4576 




Dep Mean 




2.62818 Adj 


R-sq 


0.4571 




C. V. 


23.87866 









Parameter Estimates 







Parameter 


Standard 


T for HO: 




Variable 


DF 


Estimate 


Error 


Parameter=0 


Prob > |T| 


INTERCEP 


1 


0.835320 


0.05701109 


14 .652 


0.0001 


GPA 


1 


0.563817 


0.04456166 


12.653 


0.0001 


GPA_FR 


1 


-0.127238 


0.02141033 


-5.943 


0.0001 


GPA_2 


1 


0.049450 


0.00876481 


5.642 


0.0001 


GPA_FR_2 


1 


0.024989 


0.00713693 


3.501 


0.0005 


GPA_SNGL 


1 


-0.039754 


0.03252106 


-1.222 


0.2216 


GPA_MULT 


1 


-0.241102 


0.08480797 


-2.843 


0.0045 


D_SNGL 


1 


0.118288 


0.07513378 


1.574 


0 .1154 


D_MULT 


1 


0.813128 


0.16493847 


4.930 


0.0001 


ENGG 


1 


-0.067416 


0.01350199 


-4.993 


0.0001 


FEMALE 


1 


0.093508 


0.01157382 


8.079 


0.0001 


AFRI AM 


1 


-0.204062 


0.02765399 


-7.379 


0.0001 



Table 17 



Regression Analysis with Single- vs. Multiple-Program Students 
Freshman Sample Model 



Source 
Model 
Error 
C Total 





Analysis of 


Variance 








Sum of 


Mean 






DF 


Squares 


Square 


F Value 


Prob>F 


9 


1809.42734 


201.04748 


520.360 


0.0001 


4652 


1797.35807 


0.38636 






4661 


3606.78541 









Root 


MSE 


0.62158 


R-square 


0.5017 




Dep 


Mean 


2.46603 


Adj R-sq 


0.5007 




C. V. 




25.20567 












Parameter Estimates 










Parameter 


Standard 


T for HO: 




Variable 


DF 


Estimate 


Error 


Parameter=0 


Prob > |T| 


INTERCEP 


1 


0.925132 


0.07114648 


13.003 


0.0001 


GPA 


1 


0.364218 


0.05843510 


6.233 


0.0001 


GPA_2 


1 


0.086616 


0.01183907 


7.316 


0.0001 


GPA_SNGL 


1 


0.000350 


0.03663825 


0.010 


0.9924 


GPA_MULT 


1 


-0.225065 


0.08516368 


-2.643 


0 . 0083 


D_SNGL 


1 


0.017413 


0.08624068 


0.202 


0.8400 


D_MULT 


1 


0.811986 


0.16827853 


4.825 


0.0001 


ENGG 


1 


-0.099747 


0.02197652 


-4.539 


0.0001 


FEMALE 


1 


0.146835 


0.01945596 


7.547 


0.0001 


AFRI AM 


1 


-0.293894 


0.04477054 


-6.564 


0.0001 



o 
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